Description

Implemented SharedProcessPool and MultiProcessingStage.

SharedProcessPool

• A singleton process pool that manages tasks from multiple stages, each with its own queue. Allows setting stage-specific process usage to cap the number of workers a stage can utilize.
• The total number of workers is determined by the environment variable SHARED_PROCESS_POOL_CPU_USAGE, which is multiplied by the system’s available CPU cores (os.sched_getaffinity(0)).
  • If the variable is not set, the default is 50% of available CPUs.
• Accepts process function with arbitrary argument signatures.

MultiProcessingStage

• A versatile base stage that uses SharedProcessPool to handle incoming messages in a multiprocessing manner.
• Accepts processing functions with a single argument, capable of handling dynamic input and output types. The types are deducted at runtime based on the provided processing function.
• Two ways to use the stage:
  • Extend MultiProcessingBaseStage and implement the data processing logic in the _on_data() method.
  • Use MultiProcessingStage.create() by passing in the processing function directly.
• Able to run multiple instances of the stage in the same pipeline.

Closes #1850

By Submitting this PR I confirm:

• I am familiar with the Contributing Guidelines.
• When the PR is ready for review, new or existing tests cover these changes.
• When the PR is ready for review, the documentation is up to date with these changes.